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ABSTRACT 

In  order  to  extend  the  usefulness  of  small  unmanned 
ground  vehicles  (UGVs)  to  a  wider  range  of  missions,  we 
are  developing  techniques  to  enable  high-speed 
teleoperated  control.  Our  goal  is  to  quadruple  the  speed 
of  teleoperated  UGVs  compared  to  currently  deployed 
models.  The  key  limitation  is  not  mechanical,  but  in  the 
capability  of  the  operator  to  maintain  situational 
awareness  and  control  at  higher  speeds.  To  address  these 
issues,  we  are  developing  technologies  for  immersive 
teleoperation  and  driver-assist  behaviors.  Our  immersive 
teleoperation  system  uses  a  head-mounted  display  and 
head-aimed  cameras  to  provide  the  operator  with  the 
illusion  of  being  in  the  vehicle  itself.  Driver-assist 
behaviors  will  reduce  the  cognitive  load  on  the  operator 
by  automatically  avoiding  obstacles  while  maintaining  a 
specified  heading  or  following  a  building  wall  or  street. 
We  have  demonstrated  immersive  teleoperation  on  the 
iRobot  Warrior  UGV  and  a  high-speed  surrogate  UGV. 
In  the  near  future,  we  will  integrate  driver-assist 
behaviors  and  improved  immersive  teleoperation  on  a 
high-speed  Warrior  and  test  this  system  in  realistic 
MOUT  environments. 

1.  INTRODUCTION 

Small  UGVs  such  as  the  iRobot  PackBot  have 
revolutionized  the  way  in  which  soldiers  fight  wars.  A 
typical  UGV  transmits  video  from  an  onboard  camera 
back  to  the  operator  control  unit  (OCU)  that  displays  the 
video  on  a  computer  screen.  In  a  manner  similar  to 
playing  a  first-person  shooter  video  game,  the  operator 
teleoperates  the  UGV  using  a  joystick,  gamepad,  or  other 
input  device  to  control  vehicle  motion.  While  this 
teleoperation  method  works  well  at  slow  speeds  in  simple 
environments,  viewing  the  world  through  a  fixed  camera 
limits  the  operator’s  situational  awareness  about  the 
environment.  Even  joystick-controlled  cameras  that  pan 
and  tilt  can  be  distracting  to  operate  while  driving  the 
vehicle.  This  is  one  of  the  reasons  why  small  UGVs  have 
been  limited  to  traveling  at  slow  speeds  and  missions 
where  these  speeds  are  tolerable. 

Faster  small  UGVs  would  be  useful  in  a  wide  range 
of  military  operations.  When  an  infantry  squad  storms  a 


building  held  by  insurgents,  speed  is  essential  to  maintain 
the  advantage  of  surprise.  When  a  dismounted  infantry 
unit  patrols  a  city  on  foot,  the  soldiers  need  a  UGV  that 
can  keep  up.  However,  driving  at  high  speeds  through 
complex  urban  environments  is  difficult  for  any  vehicle, 
and  small  UGVs  face  additional  challenges.  Small  UGVs 
need  to  steer  around  obstacles  that  a  larger  vehicle  could 
drive  over.  A  bump  that  would  be  absorbed  by  a  large 
vehicle’s  suspension  can  send  a  small,  fast-moving  UGV 
flying  into  the  air. 


Figure  1:  iRobot  Warrior  UGV  with  Chatten  Head- Aimed 
Remote  Viewer  (HARV) 


For  the  Stingray  Project,  funded  by  the  US  Army 
Tank- Automotive  Research,  Development  and 
Engineering  Center  (TARDEC),  iRobot  Corporation  and 
Chatten  Associates  are  developing  technologies  that  will 
enable  teleoperation  of  small  UGVs  at  high  speeds 
through  urban  terrain.  Our  approach  combines  immersive 
telepresence,  which  gives  the  operator  the  impression  of 
being  in  the  vehicle,  along  with  semi-autonomous  driver- 
assist  behaviors,  which  command  the  vehicle  to  safely 
maneuver  according  to  the  driver’s  intent.  These 
behaviors  are  similar  to  modem  fighter  aircraft,  which  are 
designed  to  be  dynamically  unstable  and  can  only  be 
controlled  through  software  that  interprets  the  pilot’s 
control  inputs. 

In  Phase  I  of  the  Stingray  Project,  we  mounted  a 
Chatten  Head- Aimed  Remote  Viewer  (HARV)  on  an 
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iRobot  Warrior  UGV  prototype  (Figure  1)  and  a  surrogate 
small  UGV  based  on  a  high-speed,  gas-powered,  radio- 
controlled  car  platform  (Figure  3).  The  operator  wears  a 
head-mounted  display  and  a  head  tracker  (Figure  2).  The 
display  shows  the  video  from  the  HARV’s  camera,  which 
is  mounted  on  a  pan/tilt/roll  gimbal.  The  HARV  tracks 
the  operator’s  head  position  and  turns  the  camera  to  face 
in  the  same  direction. 

For  Phase  II  of  Stingray,  we  will  increase  the  Warrior 
UGV’s  top  speed  by  developing  a  high-speed,  wheeled 
version  of  the  Warrior  (Figure  5).  To  assist  the  driver  in 
controlling  this  speed,  we  will  reuse  the  street-following 
and  perimeter-following  behaviors  developed  for  the 
TARDEC-funded  Wayfarer  Project.  These  behaviors  use 
LIDAR  to  determine  the  orientation  of  features  such  as 
street  boundaries,  building  walls,  and  tree  lines.  During 
Wayfarer,  we  demonstrated  that  the  performance  of  this 
system  is  robust  to  irregularities  in  both  urban  and  rural 
terrain  (Yamauchi,  2006). 

With  these  capabilities,  operators  will  be  able  to  drive 
the  Stingray  UGV  at  much  higher  speeds  than  current 
small  UGVs.  Stingray  will  be  able  to  keep  up  with 
soldiers  in  high  speed  operations  and  extend  the  tactical 
value  of  small  UGVs  to  a  wide  range  of  infantry  missions. 

2.  CHATTEN  HEAD-AIMED  REMOTE  VIEWER 

Chatten  Associates  conducted  pioneering  research  in 
high-speed  teleoperation  of  remote  ground  vehicles  under 
two  DARPA  contracts  in  the  late  1960s  and  early  1970s. 
Our  head-aimed  vision  technology  was  then  used  by  the 
US  Army  in  the  Cobra  AH-1  and  Apache  AH-64 
helicopters.  The  US  Navy’s  SPAWAR  group 
successfully  used  this  technology  to  remotely  operate 
dune  buggies  and  HMMWVs  in  the  late  1980s  and  early 
1990s. 

Testing  of  remotely  operated  ground  vehicles  on 
DARPA  and  Army  Research  Lab  (ARL)  contracts  has 
shown  that  head-aimed  vision  improves  teleoperation 
mission  performance  between  200%  and  400%, 
depending  on  the  task.  In  general,  as  the  complexity  of 
the  task  increases,  the  relative  advantage  provided  by 
head-aimed  vision  is  much  greater. 

Under  contract  to  ARDEC,  Chatten  Associates  has 
developed  the  ruggedized  HARV  suitable  for  operating  a 
small  robotic  ground  vehicle.  Previous  experiments 
(Massey,  2004)  have  shown  that  only  head-aimed  vision 
can  provide  sufficient  awareness  for  teleoperation  at  these 
speeds. 


Figure  2:  Operator  driving  vehicle  using  Chatten  HARV 

Figure  2  shows  a  remote  operator  driving  a  vehicle 
using  the  HARV.  The  operator  wears  a  head-mounted 
display  that  shows  the  current  view  through  the  HARV 
cameras  and  a  head  tracker  that  monitors  the  operator’s 
head  position.  As  the  operator  turns  his  head,  the  HARV 
gimbal  automatically  turns  the  cameras  to  face  the 
corresponding  direction.  This  provides  a  far  more 
immersive  experience  than  aiming  the  camera  with  a 
joystick  (even  with  the  same  head-mounted  display). 

3.  WARRIOR/HARV  INTEGRATION 

We  successfully  integrated  the  HARV  with  a  Warrior 
UGV  prototype.  The  HARV  is  powered  by  a  DC-to-DC 
converter  that  provides  regulated  24V  from  the  Warrior’s 
unregulated  48V  system  voltage.  The  operator  wears  the 
HARV  and  drives  the  Warrior  via  joystick  control  using 
the  prototype’s  R/C  control  interface.  Analog  video  is 
transmitted  back  to  the  operator  using  a  2.4  GHz 
transmitter.  Digital  commands  to  update  the  camera 
position  based  on  the  operator’s  head  position  are 
transmitted  on  a  separate  channel  at  900  MHz. 

Using  this  setup,  we  were  able  to  successfully 
teleoperate  the  Warrior  using  the  HARV  through  both 
open  and  wooded  terrain  and  over  grass,  asphalt,  snow, 
and  concrete  curbs.  We  were  able  to  drive  the  Warrior  at 
the  prototype’s  current  maximum  speed  while  making 
turns  and  avoiding  obstacles. 

Experienced  Warrior  operators  reported  substantial 
increases  in  situational  awareness.  The  operators  were 
able  to  control  the  robot  non-line-of-sight  at  full  speed 
through  a  grove  of  trees,  without  any  difficulty.  Based  on 
post-run  operator  evaluations,  head-aiming  increased 
situational  awareness  by  nearly  an  order  of  magnitude 
over  the  fixed  camera  solution  that  had  been  used 
previously.  The  operators  felt  very  comfortable  with  full 
speed  operations  in  areas  where  they  would  have  been 
reluctant  to  otherwise  venture. 


4.  HIGH-SPEED  SURROGATE  UGV 

In  order  to  gather  experimental  data  on  the  high¬ 
speed  teleoperation  issues  during  Phase  I,  we  used  a 
surrogate  high  speed  UGV  (Figure  3).  The  surrogate 
UGV  consists  of  a  1/5  scale  radio-controlled,  gas- 
powered  Ford  GT  with  a  top  speed  of  50  mph  (80  kph). 
This  car,  a  Hobbypro  Model  H5,  is  36  inches  (92  cm) 
long  and  21  inches  (53  cm)  wide,  and  weighs  9  kg.  It  is 
powered  by  a  23 cc  engine  that  runs  on  a  mixture  of 
gasoline  and  two-stroke  oil  (25:1  ratio),  and  is  equipped 
with  four-wheel  disc  brakes.  The  car  is  controlled  by  a 
two-channel  FM  transmitter  that  provides  proportional 
control  of  steering  and  throttle/braking. 


Figure  3:  Surrogate  UGV  driving  through  slalom  course 

We  removed  the  outer  shell  of  the  car  and  integrated 
the  HARV  and  a  roll  cage  with  the  surrogate  UGV.  In 
this  configuration,  the  surrogate  was  also  able  to  reach  an 
estimated  top  speed  of  30  mph  (48  kph).  To  the  best  of 
our  knowledge,  this  surrogate  is  the  fastest  man-portable 
UGV  to  ever  be  tested. 

The  vehicle  had  three  radios:  a  75  MHz  pulse-width 
modulated  radio  for  vehicle  steering,  throttle,  and 
braking;  a  1.7  GHz  analog  radio  for  NTSC  video;  and  a 
2.4  GHz  digital  radio  for  gimbal  control.  The  75  MHz 
radio  had  a  maximum  range  of  about  250  feet  (76  m). 

We  tested  the  surrogate  UGV  on  a  slalom  course 
(Figure  4)  shows  the  layout  of  the  course,  using  16  traffic 
cones  on  a  large,  flat  asphalt  surface,  measuring 
approximately  40  x  80  feet  (12  x  24  m).  The  yellow 
circles  in  this  figure  represent  cones,  and  the  arrows 
indicate  the  car’s  direction  of  travel  when  passing  through 
the  gate  between  the  adjacent  cones. 

In  all  of  our  experiments,  the  operators  wore  the 
HARV  head-mounted  display  and  controlled  the  vehicle 
using  the  hand-held  R/C  proportional  controller  (with  a 
small  wheel  controlling  steering  and  a  trigger  controlling 
throttle  and  brakes).  In  the  first  set  of  experiments,  the 
gimbal  was  locked  down  so  that  its  orientation  was  fixed 


with  respect  to  the  vehicle.  In  the  second  set  of 
experiments,  the  HARV  gimbal  was  enabled,  and  the 
cameras  orientation  was  controlled  by  the  operator’s  head 
motions. 


Figure  4:  High-speed  slalom  course 
(yellow  circles  indicate  cones;  arrows  indicate  vehicle 
direction  through  gates) 


Time 

Cones  Hit 

Gates  Missed 

Operator  1 

28.7 

1 

1 

28.3 

0 

0 

26.2 

0 

0 

Operator  2 

39.7 

0 

0 

34.1 

0 

0 

36.1 

2 

0 

Table  1:  Results  from  high-speed  slalom  experiments 
using  fixed  camera 


Time 

Cones  Hit 

Gates  Missed 

Operator  1 

29.6 

0 

1 

30.5 

2 

0 

28.5 

0 

0 

Operator  2 

34.2 

0 

0 

33.8 

1 

1 

49.4* 

0 

2 

Table  2:  Results  from  high-speed  slalom  experiments 
using  head-aimed  camera 

(*  =  battery  low,  resulting  in  loss  of  control  authority) 


Our  experiments  showed  that  both  operators  were 
able  to  successfully  navigate  the  slalom  course  at  high 
speed  using  the  HARV  in  either  fixed  or  head-aimed 
mode.  Table  1  and  Table  2  show  the  results  from  these 
experiments.  Operator  1  was  more  experienced  with  high¬ 
speed  driving  of  full-sized  automobiles  and  did  slightly 
better  with  the  fixed  camera.  Operator  2  did  not  have  the 


same  background  in  high-speed  driving  at  full-scale,  and 
he  drove  more  slowly  in  both  conditions  than  Operator  1 . 
Operator  2  appeared  to  benefit  somewhat  from  head¬ 
aiming,  as  it  gave  him  more  confidence  in  turning  into  the 
next  set  of  cones. 

In  a  course  that  more  closely  resembles  an  urban 
environment,  with  limited  line-of-sight  and  the  need  to 
look  around  corners,  other  experiments  have  shown  that 
head-aiming  results  in  a  substantial  improvement  in 
operator  driving  performance  versus  a  fixed  camera.  We 
expect  that  when  we  test  the  Stingray  UGV  in  MOUT 
environments,  head-aiming  will  be  crucial  for  high-speed 
control. 

5.  HIGH-SPEED  STINGRAY  WARRIOR 

In  Phase  II,  we  will  modify  the  Stingray  UGV  to 
increase  its  maximum  speed.  The  Warrior’s  standard 
drive  train  is  designed  to  maximize  torque  in  order  to 
maximize  payload  capacity.  Maximum  torque  comes  at 
the  cost  of  reduced  top  speed.  During  Phase  II  we  will 
replace  the  high  torque  gears  with  high  speed  gears. 

At  high  speeds,  the  high  traction  of  the  Warrior’s 
treads  becomes  a  liability,  reducing  vehicle  speed  and 
increasing  power  drain.  Detracking  -  tracks  separating 
from  the  drive  wheels  -  also  becomes  a  possibility  at  high 
speeds.  For  these  reasons,  our  second  modification  will 
be  to  replace  the  Warrior’s  tracks  and  flippers  with 
wheels.  This  should  substantially  increase  the  maximum 
effective  speed  of  the  vehicle. 


Figure  5:  High-speed  Stingray  Warrior  UGV 

The  tradeoff  is  reduced  ability  to  climb  large 
obstacles  and  stairs.  Without  the  flipper,  the  maximum 
forward  offset  of  the  articulated  platform  will  also  be 
reduced.  However,  we  expect  that  the  dynamic  weight 
shifting  behavior  will  be  actually  be  more  effective  in 
altering  handling  characteristics,  due  to  the  reduced  size 
of  the  ground  contact  patch  for  wheels  (vs.  tracks). 


6.  DRIVER  ASSIST  BEHAVIORS 

We  will  develop  a  set  of  semi-autonomous  driver- 
assist  behaviors  to  help  the  operator  control  the  Stingray 
UGV  at  high  speeds.  These  behaviors  will  take 
commands  from  the  OCU,  process  data  from  the  sensors 
(LIDAR,  GPS,  INS),  and  send  motion  commands  to  the 


Figure  6:  Wayfarer  PackBot  performing  autonomous 
perimeter  following 


We  will  be  able  to  leverage  and  reuse  much  of  the 
software  we  developed  for  the  TARDEC-funded 
Wayfarer  Project  (Yamauchi,  2006).  Figure  6  shows  the 
Wayfarer  PackBot  performing  autonomous  perimeter 
following  in  an  urban  environment,  using  a  combination 
of  LIDAR  and  stereo  vision  for  detecting  walls  and 
avoiding  obstacles. 

These  behaviors  will  reduce  the  cognitive  load  on  the 
operator  and  allow  the  operator  to  focus  attention  on  the 
other  tasks  using  the  HARV.  For  example,  the  operator 
could  order  the  UGV  to  drive  down  a  street,  while  the 
operator  scans  the  environment  for  potential  threats.  We 
expect  this  division  of  labor  to  greatly  improve  operator 
performance  in  reconnaissance  and  search  tasks. 

7.  OBSTACLE  AVOIDANCE 

We  will  be  reusing  the  obstacle  avoidance  behavior 
from  the  iRobot  Mapping  Kit.  This  behavior  uses  a 
randomized  Monte  Carlo  algorithm  for  generating 
potential  paths  and  then  evaluates  these  paths  based  on 
behavior  input  and  obstacle  detection.  The  behavior 
provides  a  guarded  motion  capability  that  steers  the 
vehicle  in  the  direction  commanded  by  the  operator,  if 
possible,  but  also  steers  around  any  obstacles  in  its  path. 

In  addition,  this  behavior  can  be  integrated  with 
autonomous  capabilities  (such  as  perimeter  and  street 
following).  In  autonomous  mode,  the  other  behaviors 


supply  desired  motion  commands,  and  the  obstacle 
avoidance  behavior  attempts  to  follow  these  commands  as 
closely  as  possible,  while  also  swerving  around  any 
obstacles  in  its  path. 

8.  PERIMETER  AND  STREET  FOLLOWING 

The  perimeter  and  street  following  behaviors  use  the 
Hough  Transform  to  detect  the  location  and  orientation  of 
walls  and  street  features.  The  Hough  Transform  is  a 
computer  vision  technique  that  works  by  transforming 
image  point  coordinates  into  votes  in  the  parameter  space 
of  possible  lines.  Each  point  corresponds  to  a  vote  for  all 
of  the  lines  that  pass  through  that  point.  By  finding  the 
strongest  points  in  the  parameter  space,  the  Hough 
Transform  can  determine  the  parameterized  equations  for 
the  strongest  lines  in  the  image. 

The  Hough  Transform  processes  range  data  from  the 
LIDAR  and  calculates  the  strongest  line  orientations  and 
offsets  relative  to  the  robot’s  current  position.  Our 
experiments  showed  that  the  system  is  highly  accurate 
and  reliable  in  determining  the  location  and  orientation  of 
walls  both  outdoors  and  indoors. 

We  interfaced  the  Hough  Transform  line  detector 
with  a  perimeter  following  behavior.  The  follow- 
perimeter  behavior  attempts  to  steer  the  robot  so  that  it  is 
parallel  to  the  strongest  line  detected  by  the  Hough 
Transform.  To  prevent  the  robot  from  oscillating  between 
two  lines  that  are  approximately  the  same  strength,  an 
accumulator  array  is  used  to  integrate  the  strength  of  line 
orientations  over  time.  For  computational  efficiency,  all 
lines  with  the  same  orientation  vote  for  the  same 
orientation,  regardless  of  the  range  from  each  line  to  the 
robot.  Orientations  are  grouped  into  5  degree  bins  for  a 
total  of  72  bins. 


Figure  7:  Hough  Transform  finds  perimeter  orientation 

The  follow-perimeter  behavior  outputs  a  desired 
absolute  heading  in  world  coordinates.  This  desired 


heading  is  passed  to  the  obstacle  avoidance  system,  which 
selects  the  obstacle-free  heading  that  is  closest  to  the 
desired  heading  output  by  follow -perimeter.  This  allows 
the  robot  to  reactively  steer  around  obstacles  that  are 
located  next  to  walls  and  then  resume  wall-following 
automatically. 

Our  experiments  showed  that  the  Hough  Transform 
works  well  to  detect  perimeter  orientation  even  when  the 
laser  plane  does  not  directly  intersect  the  building  wall. 
Figure  7  shows  the  Wayfarer  PackBot  following  the 
raised  landscape  behind  iRobot  HQ.  The  yellow 
rectangle  represents  the  robot’s  current  position;  the 
purple  dots  indicate  LIDAR  returns;  and  the  blue  line 
shows  the  wall  orientation  as  determined  by  the  Hough 
Transform.  In  this  example,  the  laser  plane  intersects  the 
grass  and  dirt  of  the  landscape  instead  of  the  building 
wall,  so  the  returned  points  are  not  perfectly  aligned. 
Despite  this,  the  Hough  Transform  is  able  to  accurately 
determine  the  heading  of  the  building  perimeter. 

The  Wayfarer  follow-street  behavior  is  similar  to  the 
follow-perimeter  behavior,  but  tracks  lines  on  both  sides 
of  the  robot. 

9.  DYNAMIC  WEIGHT  SHIFTING 

The  Warrior  can  radically  change  the  position  of  its 
center  of  mass  by  moving  an  articulated  platform  forward 
and  backwards.  We  will  study  the  effect  of  platform 
position  on  the  Warrior’s  handling.  We  will  evaluate  a 
range  of  different  configurations  in  the  range  between  the 
maximum  backward  and  maximum  forward  positions.  It 
is  likely  that  the  maximum  forward  position  is  unstable 
for  the  wheeled  UGV,  so  we  will  determine  the  maximum 
stable  forward  position  of  the  platform  on  this  vehicle. 

When  a  vehicle  oversteers,  its  rear  tires  lose  traction 
before  the  front  tires,  causing  the  vehicle  to  turn  more 
than  commanded.  Vehicles  with  more  weight  on  the  rear 
wheels  tend  to  oversteer  more  easily.  When  a  vehicle 
understeers  its  front  tires  lose  traction  before  the  rear  tires, 
causing  the  vehicle  to  turn  less  than  commanded. 
Vehicles  with  more  weight  on  the  front  wheels  tend  to 
understeer  more  easily.  Vehicles  that  have  roughly  equal 
weight  over  both  axles  tend  to  have  more  neutral 
handling,  meaning  they  that  are  less  likely  to  oversteer  or 
understeer,  and  that  either  oversteer  or  understeer  can  be 
triggered  based  on  control  inputs. 

Understeer  is  almost  always  undesirable.  However,  a 
skilled  driver  can  use  oversteer  to  take  turns  faster  than 
otherwise  possible.  The  driver  intentionally  causes  the 
rear  wheels  to  break  free,  steers  toward  the  exit  point  of 
the  curve,  and  hits  the  throttle  to  break  out  of  the  slide. 
Done  properly,  this  allows  the  vehicle  to  maintain  more 
speed  through  the  turn. 


We  will  explore  whether  controlled  oversteer  can  be 
used  to  increase  the  turn  rate  beyond  what  can  be 
achieved  without  wheel  slip.  For  these  experiments,  we 
will  put  the  vehicle  in  the  configuration  that  maximizes 
oversteer,  and  then  we  will  determine  whether  we  can 
take  turns  faster  and/or  sharper  in  this  mode. 

The  results  from  these  experiments  will  be  used  to 
design  the  dynamic  handling  behaviors  for  the  Stingray 
UGV.  These  results  will  tell  us  what  track  and  weight- 
balance  configurations  we  should  use  for  maximum  speed 
in  a  straight  line,  for  maximum  turn  rate,  for  maximum 
speed  through  turns,  and  for  controlled  oversteer.  These 
behaviors  will  alter  the  Warrior  platform  position  on  the 
fly  based  on  driver  input. 

For  example,  if  the  driver  is  commanding  maximum 
speed  in  a  straight  line,  the  dynamic  handling  behaviors 
will  put  the  vehicle  into  the  configuration  that  was 
determined  to  maximize  straight  line  speed.  If  the  driver 
is  commanding  a  hard  right  turn  or  hard  left  turn  at  low 
speed,  the  behaviors  will  select  the  configuration  that 
minimizes  the  vehicle  turn  radius.  However,  if  the  driver 
is  commanding  a  hard  turn  at  high  speed,  the  behaviors 
will  select  the  configuration  that  maximizes  speed 
through  a  sharp  turn. 

10.  HEADING  CONTROL 

A  small,  lightweight  UGV  that  drives  at  high  speeds 
will  spend  a  significant  proportion  of  time  in  the  air.  This 
is  especially  true  in  rough  off-road  terrain,  but  even  in 
urban  environments,  uneven  road  surfaces  may  result  in  a 
small,  fast-moving  UGV  catching  “air  time”.  Our 
previous  experience  at  both  iRobot  and  Chatten  with 
driving  lightweight  UGVs  at  high  speeds  shows  that  this 
is  unavoidable  for  a  vehicle  with  the  weight  (less  than  100 
kg)  and  speed  (up  to  40  kph)  requirements  for  this  project, 
unless  it  has  a  suspension  that  would  add  substantial 
weight,  cost,  and  complexity  to  the  vehicle  platform.  So 
instead  of  adding  a  heavy,  expensive  suspension  to  the 
small  UGV  platform,  we  will  focus  on  maintaining 
control  of  the  vehicle  at  high  speeds  despite  the  frequent 
loss  of  contact  with  the  ground. 

Our  heading  control  behavior  will  allow  the  operator 
to  specify  a  desired  heading  for  the  vehicle  and  command 
the  vehicle  to  maintain  that  heading  at  a  specified  speed. 
We  expect  this  to  be  particularly  useful  at  high  speeds 
when  continuous,  rapidly-changing  steering  inputs  are 
required  to  maintain  the  vehicle  heading  if  the  vehicle 
loses  contact  with  the  ground  and  turns  in  the  air.  Like 
the  perimeter  following  and  street  following  behaviors, 
the  heading  controller  will  free  the  operator  to  focus 
attention  on  the  surrounding  environment  instead  of  being 
distracted  by  driving  the  vehicle. 


11.  ENHANCED  SITUATIONAL  AWARENESS 

Based  on  Phase  I  developmental  work,  it  is  clear  that 
adding  additional  cameras  as  inset  views  into  the  main 
image  of  a  head-aimed  vision  system  can  greatly  increase 
situational  awareness.  Figure  8  (top)  shows  a  360° 
panoramic  overlay  on  the  main  view.  Figure  8  (bottom) 
shows  a  rear- view  camera  overlay  on  the  main  view. 

With  the  360°  overlay,  any  motion  in  the  panoramic 
image  that  is  inconsistent  with  the  motion  of  the 
remainder  of  the  panoramic  view  will  instinctively  cue  the 
operator's  peripheral  view  processing  to  turn  and  look  at 
the  movement.  Presenting  image  data  this  way  to  the 
operator  takes  maximum  advantage  of  the  operator's 
native  visual  processing.  All  of  us  instinctively  process 
peripheral  vision  and  automatically  turn  to  look  at 
movement  in  our  periphery. 

Similarly,  the  rear-view  camera  functionality  takes 
advantage  of  operators'  long-term  experience  with  driving 
cars.  Operators  reversing  the  robot  out  of  a  tight  passage 
will  be  able  to  use  the  main  display  to  constantly  look  left 
and  right  to  check  side  clearances,  while  using  the  inset 
image  from  the  fixed  rear-facing  camera  to  gauge 
alignment  and  rear  clearances. 

We  will  modify  our  HARV  to  add  five  very  small 
video  cameras,  of  the  type  used  in  cell  phones.  Four  of  the 
cameras  will  be  mounted  at  90°  spacing  around  the 
perimeter  of  the  gimbal  on  the  pan  yoke  ring,  which 
moves  with  the  pan  axis.  These  four  cameras  will  always 
have  a  fixed  relationship  with  viewing  angle  of  the  main 
display.  What  is  shown  to  the  left  in  the  panoramic  image 
will  be  to  the  left  of  where  the  HARV  is  currently 
looking,  and  visa-versa.  The  fifth  camera  will  be 
mounted  in  the  base  tube.  This  camera  will  face  to  the 
rear  and  slightly  down,  to  provide  a  good  view  for 
backing  up. 

We  will  use  low-power  video  processing  electronics 
to  merge  the  four  90°  cameras  together  to  create  the 
panoramic  image,  and  then  overlay  it  on  the  NTSC  video 
stream,  without  inducing  any  latency  into  the  main  video 
image.  We  will  do  the  same  thing  for  the  single  back-up 
camera  image.  We  will  also  create  a  simple  operator 
interface  for  selecting  the  different  view  modes. 

Because  high-speed  teleoperation  is  so  dependent  on 
effective  human-robotic  interaction,  testing  with  Army 
soldiers  will  be  an  important  component  of  our 
development  process.  We  plan  to  have  a  testable 
prototype  of  the  enhanced  situational  awareness  HARV 
early  in  Phase  II,  so  that  we  can  conduct  testing  of  high¬ 
speed  reconnaissance  at  the  McKenna  MOUT  site  at  Fort 
Benning  in  2009. 


Figure  8:  HARV  main  view  with  panorama  (top) 
and  rear  view  camera  (bottom) 


This  early  testing  will  be  a  quick  reality  check, 
looking  at  the  issue  of  how  soldiers  might  make  use  of  a 
high-speed  robotic  reconnaissance  platform,  and  what 
added  functionality  would  be  important  to  them.  We  will 
specifically  get  data  and  feedback  on  our  enhanced 
situational  awareness  prototype. 

12.  LOW-LATENCY  DIGITAL  VIDEO 

Vision  is  a  critical  teleoperation  interface,  but  video 
requires  very  high  communications  bandwidth.  In 
addition,  video  latency  can  critically  impact  mission 
performance  and  reduce  operator  effectiveness.  The 
general  figure  of  merit  in  the  automotive  simulation 
community  is  that  visual  latency  needs  to  be  below 
100  ms  in  order  for  a  car  to  be  controllable. 

Driving  performance  degrades  with  any  measurable 
latency,  and  the  level  of  performance  impact  gets  steadily 
worse  as  latency  increases.  For  this  reason,  we  have  used 
analog  video  and  analog  radios  to  transmit  the  video. 
However,  the  future  of  video  transmission  in  the  military 


is  digital,  so  we  will  develop  a  digital  protocol  for  our 
HARV  system. 

In  Phase  II,  we  will  work  within  the  existing  H.264 
digital  video  codec  to  create  a  low-latency,  high-quality, 
low-bandwidth  transmission  protocol  that  is  compliant 
with  the  H.264  spec  and  with  military  IP  data  radios.  The 
video  codec  will  use  a  technique  that  has  been  in  the 
specification  since  H.261,  but  has  never  been 
implemented,  except  in  academic  research  settings.  This 
technique  is  sometimes  called  ’’distributed  keyframing”. 

In  digital  video,  a  keyframe  is  a  single,  complete 
picture  of  a  video  frame,  compressed  in  a  similar  fashion 
to  a  JPEG  photo.  Low  latency  digital  video  sends  every 
video  frame  as  a  keyframe.  As  long  as  there  is  sufficient 
bandwidth,  the  result  is  good  quality,  high  frame  rate,  and 
low  latency.  Military  IP  data  radios,  however,  lack  the 
bandwidth  needed  for  this  type  of  compression.  The 
video  is  reduced  to  very  low  quality  and  low  frame  rates. 
The  reduction  in  frame  rate  alone  can  cause  the  latency  to 
exceed  100  ms. 

Time  and  motion  based  compression  schemes,  such 
as  MPEG-4,  use  P-frames  that  contain  only  incremental 
changes  from  the  previous  frame.  A  video  stream  can 
have  the  same  apparent  quality  as  the  original  analog 
image  using  periodic  keyframes  along  with  P-frames  that 
are  only  5%  to  10%  of  the  size  of  a  keyframe  (also  called 
an  I-frame).  In  high  motion  video,  there  may  be  one 
keyframe  every  second,  which  accounts  for  about  half  the 
bandwidth.  The  problem  is  that  is  a  bandwidth- 
constrained  environment,  the  keyframe  would  take  a  half 
second  to  transmit,  causing  at  least  a  500  ms  latency  in 
the  video  image. 

The  distributed  keyframe  technique  breaks  a  single 
keyframe  up  into  groups  of  ’’macroblocks”.  If  there  are 
300  macroblocks  in  a  video  frame  and  30  frames  per 
second,  then  keyframe  data  for  10  macroblocks  would  be 
sent  along  with  P-frame  data  for  the  other  290 
macroblocks  in  each  video  frame.  The  result  is  an  entire 
new  keyframe  each  second,  but  only  17  ms  of  latency 
(due  to  keyframe  transmission  delays). 

There  are  a  number  of  other  techniques  that  are  also 
unimplemented  in  commercial  adaptations  of  the  H.264 
specification  that  would  help  teleoperation  in  a 
bandwidth-constrained  environment.  The  quality  factor 
for  the  macroblocks  in  the  center  of  the  image  can  be  set 
higher  than  in  the  periphery.  Human  vision  works  this 
way  to  reduce  internal  processing  requirements  in  our 
visual  cortex.  By  biasing  quality  to  the  center  of  the 
frame,  the  video  bandwidth  can  be  cut  in  half,  without  a 
perceptual  difference  to  the  operator.  This  is  especially 
true  when  using  head-aiming,  where  the  operator 
naturally  keeps  objects  of  interest  in  the  center  of  vision. 


In  macroblocks  within  the  frame  where  there  is  motion 
that  is  counter  to  the  overall  image  motion,  the  quality 
factor  can  be  locally  increased,  which  will  make  this 
moving  object  have  sharper  detail. 

We  can  also  make  the  video  signal  more  robust  and 
resistant  to  the  types  of  signal  drop-outs  experienced  in 
the  field.  The  video  can  be  double  encoded,  using  a 
second  low-resolution  version  behind  the  high-resolution 
video.  If  macroblocks  within  the  high-resolution  video 
are  lost,  then  the  low  resolution  will  show  through  in 
those  spots  until  new  keyframe  macroblocks  can  be  used 
to  fill  in  the  detail. 

When  IP  packets  are  dropped,  we  can  avoid 
retransmitting  the  old  packet,  which  can  cause  bottlenecks 
in  the  IP  transmission.  Instead,  we  can  request  an  updated 
P-frame  macroblock  that  skips  over  the  lost  data  and 
provides  the  incremental  changes  from  the  last  good  data. 
When  that  P-frame  data  gets  large,  we  can  provide  new 
keyframe  data  for  the  macroblocks  instead. 

Using  these  methods,  our  new  digital  protocol  will 
enable  the  low-latency,  high-quality,  low-bandwidth 
video  needed  for  immersive  teleoperation. 

CONCLUSIONS 

The  Stingray  Project  is  developing  techniques  for 
high-speed  teleoperation  of  small  UGVs.  These 
techniques  include  immersive  teleoperation  and  semi- 
autonomous  driver-assist  behaviors.  In  Phase  I  of  this 
project,  we  integrated  the  Chatten  Associates  HARV  with 
the  iRobot  Warrior  UGV  prototype,  and  were  able  to 
successfully  drive  the  UGV  through  wooded  terrain  using 
immersive  teleoperation.  We  also  performed  experiments 
with  a  high-speed  man-portable  surrogate  UGV  capable 
of  speeds  up  to  30  mph  (48  kph).  Using  immersive 


teleoperation,  we  were  able  to  drive  this  UGV  at  high 
speeds  through  a  slalom  course. 

In  Phase  II,  we  will  develop  a  high-speed  version  of 
our  Warrior.  We  will  develop  semi-autonomous  driver- 
assist  behaviors  for  obstacle  avoidance,  perimeter 
following,  street  following,  dynamic  weight  shifting,  and 
heading  control.  We  will  improve  the  HARV’s 
situational  awareness  with  additional  cameras  providing 
360°  degree  panoramic  views  and  rear- view  insets.  We 
will  also  develop  a  low-latency,  high-quality,  low- 
bandwidth  digital  video  protocol  for  transmitting  HARV 
video. 

Finally,  we  will  integrate  all  of  these  components  to 
enable  high-speed  teleoperation  of  the  modified  Warrior 
in  urban  environments.  We  will  test  this  system  in 
MOUT  environments  and  solicit  feedback  from 
warfighters  on  the  best  way  to  utilize  these  capabilities  to 
meet  current  and  future  operational  needs. 
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